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Abstract 

The star-shaped A-coalescent and corresponding A-Fleming-Viot process 
where the A measure has a single atom at unity are studied in this paper. 
The transition functions and stationary distribution of the A-Fleming-Viot 
process are derived in a two-type model with mutation. The distribution of 
the number of non-mutant lines back in time in the star-shaped A-coalescent 
is found. Extensions are made to a model with d types, either with parent 
independent mutation or general Markov mutation, and an infinitely-many- 
types model when d —?• oo. An eigenfunction expansion for the transition 
functions is found which has polynomial right eigenfunctions and left eigen¬ 
functions described by hyperfunctions. A further star-shaped model with 
general frequency dependent change is considered and the stationary dis¬ 
tribution in the Fleming-Viot process derived. This model includes a star¬ 
shaped A-Fleming-Viot process with mutation and selection. In a general 
A-coalescent explicit formulae for the transition functions and stationary 
distribution when there is mutation are unknown, however in this paper 
explicit formulae are derived in the star-shaped coalescent. 
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1. Introduction 


A Fleming-Viot process {^t}i>o representing the frequency of type 1 indi¬ 
viduals in a population of two types has a generator C° acting on functions 
in C^([0,1]) described by 


C 


'g{x)= [ x{g{x{l-y)+y)-g{x))+{l-x){g{x{l-y))-g{x))^^^^, (1) 

Jo y 


where A is a hnite measure on [0,1]. The population is partitioned at events 
of change by choosing a type 1 individual with probability x, or a type 2 in¬ 
dividual with probability 1 — x; adding in an additional frequency of y of the 
type chosen, then rescaling the frequencies to add to 1. The rate of repro¬ 
duction events with offspring frequency y is y~‘^A{dy). If A has a single atom 
at 0, t hen {C{t)}t>o is a Wright-Fisher diffusion process. iBirkner and Blath 
( 2 OO 9 I ) describe the A-Flem ing-Viot process and discre te models whose limit 
gives rise to the process. Eldon and Wakelev ( 2006l ~) introduced a model 
where A has a single point of increase in (0,1] with a possible atom at zero 
as well. A model where there is mutation from type 2 individuals to type 1 
individuals at rate 9i/2 and from type 1 to type 2 individuals at rate 82/2 
(with 6 = 61 + 62 ) has an additional term added to the generator ([1]); 


Cg{x) = C°g{x) + -{81 - 8 x)g'{x). 


( 2 ) 


A stationary distribution exists in this process because of recurrent mutation 
between types. 

The A-coalescent is a random tree back in time which has multiple merger 
rates for a specific set of 2 < k < n edges merging while n edges in the tree 
of 


^nk — 


f 


x 


■(l-x) 


^-kKdx) 


k> 2 . 


( 3 ) 


After coalescence there are n — k + 1 edges in the tr e e. The A - coales c ent pro 


Gnedin et. al. 


cess w as introduced bv lDonnellv and Kurta (1199911 : 
( I 999 II and has been extensively studied ( Pitrnanl . 


Pitman 

(1999); 

Saeitov 

2002 

: Berestvcki. 2009; 


2014 : Tellier and Lemaire . 2014l l. The coalescent process is 
a moment dual to the A-Fleming-Viot process. That is, the A-coalescent 
tree describes the genealogy of a collection of individuals ba ck in time in 
the A-Fleming-Viot process. See for example Etheridge ( 20121 ). In a model 
with mutation and generator ([2]) mutations occur at random on the edges 
of the tree. The A-coalescent is said to come down from infinity if there is 
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an enterance boundry at infinity. Then there is a finite number of edges at 
any time back in an i nitial tree beginning with an infinite number of leaves. 
Bertoin and Le Gall ( 2006l l showed that the A-coalescent comes down from 


infinity under the same condition that the continuous state branching pro¬ 
cess becomes extinct in finite time, that is when 



< oo, 


(4) 


where the Laplace exponent 



l + 92 /)y ^A((i?/). 


(5) 


GriffithsI (j2014l l shows that ([T|) can be written as a Wright-Fisher generator 
equation 

( 6 ) 


C°g{x) = -x{l - x)E yg” (x(l -W) + VW) 


where W = UY, Y has distribution A (assuming without loss of generality 
that A is a probability measure), U has a density 2u, u G {0,1), V is uniform 
on (0,1), and U,V,Y are independent. If VF = 0 the usual Wright-Fisher 
generator is obtained. 

In this paper we consider a star-shaped A-coalescent tree and Fleming- 
Viot process when the measure has a single atom at 1, with A({1}) = 1. This 
seems a simple model, however there are a number of aspects in the analysis 
that are not at all simple. In a general A-coalescent explicit formulae for 
the transition functions and stationary distribution when there is mutation 
are unknown, however in this paper explicit formulae are derived in the 
star-shaped coalescent. 

This A-coalescent does not come down from infinity because a tree be¬ 
ginning with an infinite number of leaves still contains edges joining the 
leaves until an exponentially distributed time of rate 1 when all edges coa¬ 
lesce. The generator of the star-shaped Fleming-Viot process with mutation 
included is 


Cg{x) = x{g{l) - g{x)) + {I - x){g{0) - g{x)) + -{Oi - ex)g'{x). (7) 


In this model the entire population is replaced in a reproduction event. 

A biological motivation for studying the star-shaped coalescent is that ge¬ 
nealogies in exponentally growing popula tions tend to be star-shaped. A dis- 
cussion of this property is in Section 4.2 of Durrett ( 20081 ). Griffiths and Tavar^ 
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( 19981 ) show that in an exponentially growing population with large rate /3, 
in Kingman coalescent, beginning from n individuals, is star-shaped with 
mean coalescence time asymptotically 


13 M log/?-h log 


n 


-1 


7 


where 7 = 0.577216... is Euler’s constant. Coalescence time is short with a 
large growth rate, so mutation rates have to be of order /?/ log /? for mutation 
to occur. 

In Section 2 the transition functions and stationary distribution of the 
star-shaped Fleming-Viot process are found. An interesting eigenfunction 
expansion is found for the transition functions which involves polynomial 
right eigenvectors and left eigenvectors which are described by hyperfunc¬ 
tions. The appearance of hyperfunctions is likely to be related to the fact of 
not coming down from infinity and suggests the same complexity in other 
A-coalescent processes. An expansion of the transition functions at time t 
partitioned by the number of population replacements in time t is derived. 
In Section 3 the distribution of the number of non-mutant lines back in time 
in the star-shaped A-coalescent is derived. The results in this paper are, 
to the knowled ge of the autho rs, new and not a simple application of ex¬ 
isting results in [Griffiths] (120141 ) and other authors. The model is extended 
to d types in Section 4, either with parent independent mutat ion or g enera l 


Markov mutation, and to infinitely many types as d —>• 00 . Mohlel ( 20061 ) 


found the sampling distribution in the infinitely-many-types model using 
recursive equations. We also obtain this distribution from a different argue- 
ment. In Section 4 a model with very general frequency dependent change is 
considered. This includes a star-shaped model with mutation and selection 
where the stationary distribution is obtained. 


2. The star-shaped coalescent with two types 

In this section the transition distribution and stationary distribution of the 
two-type star-shaped process with mutation is obtained. 

Theorem 1. Let {'C(t)}t>o be the frequency of type 1 individuals at time 
t, when ^(0) = x, in the Fleming-Viot process with generator ([7]). Denote 
hj = 1)2 as the probability of a change of type in a single line 
of the star-shaped coalescent in time ( 0 ,t), and qi{t]x) = xpii{t) -|- (1 — 
x)p 2 i{t) as the probability a single individual chosen at time t is of type 
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i = 1,2, assuming in both cases that there is no replacement event in (0, t). 
Specifically, with p = Oij9, 

Piiit) = + (1 - P2i{t) = (1 - 


and 

qi{t-, x) = + (1 - q 2 {t; x) = 1 - qi{t-, x). 

Then 


m = 

where in {0 


I qi{t-,x) with probability e * 

\Pii{'T) * = 1)2, with probability (l — e~^)qi{t — t;x), 
I, T has a truncated exponential distribution with density 
(l — e~*) 0 < r < t. 


( 8 ) 

( 9 ) 


That is, ^(t) only depends on the last replacement event before t at r, if a 
replacement event occurs. The continuous part of the distribution of 
has a density f{^‘,x,t) equal to 

(p + e-^^/^{x-p)^'^^(^^y j^, ^ >p+(1-p)e-®*/2 

(^1-p- e-^^/\x -p)[l- f) ”') I (l - e < P(1 - 

( 10 ) 

and zero in the region p(l — e ^ < p + (1 — p)e The stationary 

density of {^(t)}t>o is 


/(O = lim 

r—>-oo 


2 U-P 


Pe[f= 


2 / 9-1 ^ 


c>p 

^<p. 


(11) 


Proof. The formulae for pii(f), P 2 i{t), qi{t) and q 2 {t) are elementary using 
an argument that if at least one mutation occurs along single line in time 
t with probability 1 — then the line is type 1 at time t if the last 

mutation is type 1 with probability p. We use the approach of looking at the 
Lambda coalescent as a dual process to the Fleming-Viot process describing 
the population. Consider a star-shaped Lambda coalescent tree beginning 
with infinitely many leaves at time t. Complete coalescence occurs with in 
the tree at rate 1, giving an exponential time T with rate 1 to the time to the 
most recent common ancestor. Let Z{t) be the number of type 1 individuals 
in a sample of n individuals taken at time t. There are two possibilities for 
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the time of coalescence of the n individuals: T < t or T > t. The probability 
that T < t and there are z individuals of type 1 in the sample is 

W(1(12) 

*=i -^0 

and the probability that T > t and there are z individuals of type 1 in the 
sample is 

e~^(^^qiit-,xy{l - (13) 

Let ^{t) = lim^^oo The distribution of ^(t), ([8]), can be read off from 
P3D and (|12l) by using the law of large numbers in the binomial distribution. 
The first case in Q is an atom in the distribution, and the second case leads 
to a density function. In the second case there are two disjoint regions for 
the density. 

Ifpii(r) = ^ 

Ifp2i(r) = ^ 

In addition 

( 

must be satisfied in both the r 

if ^ > p 

1 - J if ^ < p. 

Evaluating the density with a change of variable r —^ from (1141) gives m- 
The limiting density as t ^ oo in (jlOp is clearly (jlip . A direct verification 
that E[£p(^)] = 0, g G ^^((0,1)) n C'([0,1]) (the space of continuous func¬ 
tions in [0,1] which have continuous second derivatives), when expectation 
is in the stationary distribution can also be shown. Applying the generator 
(j7P to g{x) and taking expectations, 

E [CgiO] =pg{l) + {l- p)g{0) - E [^(0] + [{p - 05'(0] • (15) 




then = 1 — I < p. 


(14) 


-0r/2 ^ ^-et/2 

3gions. That is, in the two regions. 
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Calculating the last term using integration by parts, 


a'im 


( 1 -: 


+ 


1 — p 


p 


, 2 /e 


{p-if^g'im 


p 


(l-p)2/e 




1 I — p 

p 




= -pg{l) - (1 - p)g{^) + E [g{C)] ■ (16) 


Substituting (fT6]l into (fTS]) shows that E[£g'(^)] =0. □ 

Throughout the paper E[-] usually denotes expectation in the stationary 
distribution, and E^, [ • ] denotes E[- | ^(0) = x]. 

Corollary 1. Moments of ^{t) are, for n = 0, 1, ... 


Ex[(^(t)-p)”] 


+ 


+ 


^-t{l+n9/2)^^_p^n 

2/0 


n + 2/0 


pii-pr + i-irii-p)p- 

I _ g-t(l+ne/2)' 


{x — p)e ®^/^(2/0) 
n-1 + 2/0 


(l-p)"-(-l)V 


1 _ g-t(l+(n-l)+2) 


(17) 


Proof. Preliminary calculations that are needed with a = 2/0, and a = 
2/0 — 1 are: 


i 


p_^(l_p)e Ot/2 
p(l_e-^V2) 


(e - pY 


^-p^a-l 1 ^^^(1-p)' 


I — p/ I — p 


n + a 


I _ g-(n+a)0t/2 


(e-p)” 1- 


a-ll 


P 


{-pY 


di= ^ 

p n + a 


1 — e 


— (n+a)dt/2 


(18) 


Considering (I18p with the different parts of the distribution of f,{t) gives 

([HI)- □ 
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Remark 1. A representation for the stationary distribution (jlip is the 
following. Let rj he a random variable with density , 0 < rj < 1. 

Then with probability p, ^ = p(l — ??) + ?? and with probability 1 — p, 

^=p{l- T]). 


2.1. Eigenvalues and Eigenvectors of the generator 


GriffithsI (j2014l L Corollary 4 shows that the right eigenvectors of the gen¬ 


eral A coalescent generator are polynomials and the eigenvalues are = 
^E[n((n — 1)(1 — -|- 9 )], where W is described in the Introduction. 

In our generator ([2]), the eigenvalues are Ai = 0/2 and for n = 2,... 


Xn = ^E[n((n-l)(l-ITr-2 + 0)] 

= 2ttn(n — 1)(1 — -|-^n0 

= l + ne/2. (19) 


Equation (I17p is consistent with right polynomial eigenvectors and eigenval¬ 
ues m because 

lEx =p+{x- 

and 

where Rn-i{x) is a polynomial of degree n — 1 in x. 

Theorem 2. The eigenvalues of £, defined in ([2]) are Ai = n6/2, Xn = 
1 +n6f2, n>2. The right eigenvectors are polynomials {Pn(ic)}5)T;^, taken 
to be monic, which satisfy 


CPn{x) = -XnPn{x). 

The explict form for these polynomials is Pi{x) = x — p, and for n > 2 

Pnix) = {x - p)'^ + Cnl{x - p) + CnO, (20) 


where 


^nO — 


Cnl — 


2/9 


n + 2/9 
2/9 


n-1 + 2/9 I 


p{l-pr + il-p)i-py 
i-pr-{i-py 


( 21 ) 

( 22 ) 
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Proof. A general Gram-Schmidt construction is the following. Suppose for 
a generator £ that Cx^ is a polynomial of degree n in x, for n G £1 = 0, 
Ao = 0, Po{x) = 1; and recursively we can write 

n—1 

£x” = -Xnx"' -h y^^ankPkjx), 

k=0 

for a set of constants {onk}- Now let 

n—1 

Pnix) = x"- CnkPkix), 

k=0 

where the constants {cnk} have to be chosen so that 

CPn{x) = -\nPn{x). 

Now 

n—1 n—1 

PPn(x) X^X -|- 'y ^ (InkPk jx') ^ ^ X^^CnkPkix) 

k=Q k=Q 

n—1 

= —Xnix p ^ ^ CnkPkjx)) . 
k=0 

Therefore we must choose —XnCnk = (ink — X^c^k or 


_ dnk 

(nk T T ) 

^k -^n 

allowing a recursive construction. In our case we are considering polynomials 
in re — p, however the idea is the same. Clearly £(x — p) = —6f2{x — p), and 
for n > 2 


C{x — p)^ = x{l — p)^ P {1 — x){—p)'^ — {x — p)^ — {x — p)^n6/2 

= -(1 n6f2){x -pT P{x- p)((l -pT - {-PT) 

pp{i -pY p {I - p){-pY. 

In the context of the recursive construction = 1 -|-n0/2, a^i = (1 — p)"" — 
(—p)"', Qno =p(l — p)” + (1 — p)(—p)""- Therefore 

(l_p)n_(_p)n 

■ Ai - A„ 

{-pT-{i-pT 

lP{n-1)6/2 ’ 

_ (I - p) (-p)” p(l - p)"- 
“ 1 + n9/2 
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and 


Pn{x) = {x -p)'^ + Cnl{x - p) + CnO- 

This agrees with (1^ . □ 

The left eigenvectors {Qn{x)}^=i are functions informally satisfying 

OO 

= f{C){l + Y.e-^-^Pn{x)Qn{0} (23) 

n=l 

with the atom included in the distribution, where /(^) is the stationary 
density ifTTI) . In fact the continuous part of the distribution has a density 

/(C;x,t) = /(0{i + e-^i‘Pi(x)gi(o} 

in the region {0 < ^ < p{l — U {p + (1 — < ^ < 1}, however 

a difficulty is the restricted support of the distribution. The formal way to 
specify an expansion which includes the atom is to require 

OO 

IE45(C(t))] = J2^~^"'^[9iOQniO]Pn{x), (24) 

71=0 

for suitable functions g G C'°°((0,1)) n ^([0,1]). The eigenfunctions have a 
biorthogonality property that 


IE[P„i(0Qn(0] =5r, 


(25) 


(5i(^) is defined in (j28jl . Although the transition density is relatively simple 
it is difficult to make either (l23l) or ((2^1) precise. The difficulty is because 
of the dust like property of the process and the way the continuous part of 
the transition density depends on t. There is an interesting solution for the 
l eft eigenvector s in terms of hyperfunctions, which are generalized functions 


( jKaneko| ,| l988| ). Denote QniO = QniOfiO'^ then E[fi((0Qn(0] = {9,Qn), 
n > 2 is regarded as an integral defined by the action of a hyperfunction 
QniO- Another difficulty is that if 9/2 < 1 then E[Qi(^)] is not well defined 
because of a singularity at p, whereas it is required that E[gf(^)(5i(^)] is 
well defined when g has a power series expansion about p. Recalling the 
representation of ^ in terms of g in Remark 1 we define 


E[Qi(0] = limE[Qi(0/{r/ > e}] = 0 
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in the sense of a Cauchy principal value. IE[(7(^)(5 i(^)] is similarly inter¬ 
preted as a Cauchy principal value. This allows (l27)l to hold for constant 
functions g. 

A hyperfunction is specified by a pair (h, k), where h and k are (complex) 
holomorphic functions defined respectively on the upper and lower complex 
half-planes. The hyperfunction is represented by its actions in complex in¬ 
tegrals on test functions with weight h — k. A technique to deal with an 
essential singularity at zero of h is to take a hyperfunction (h, h). The hyper¬ 
function we use behaves in integrals as S{^—p). We need to use derivatives of 
this function in integrals which cannot be taken in usual integration theory. 
A classical first example of a hyperfunction is how to express a Dirac delta 
function as a hyperfunction in integration. As a brief example consider the 
Dirac delta function (For simplicity the function is centered at zero.) 

Cauchy’s formula is 

= V^(0), i = v^. (26) 

Deform the contour of the integral as 


Ja+io C 27rz C 

where a and b are holomorphic points of 1/.^. The Dirac delta function as a 
hyperfunction is (l/(27ri^), l/(27ri^)). Equation (|26]) is concisely written as 
{6, ip) = ip{0), where 

Differentiation can be justified and the explicit expression for the jth. deriva¬ 
tive is 




(-l)^j! (-l)^j! 1 ^ 

’ 2 m ) ’ 


j = l,2,... 


Integration by parts gives = (—1)'? ((I, which is the Cauchy- 

Goursat formula 

2 m J j\ 

Hyperfunctions are a generalization of Schwartz distributions, which are lin¬ 
ear operators defined by their actions on test functions. Our eigenfunctions 
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could be defined by Schwartz distributions, however we prefer the elegance 
of hyperfunction theory. 

Theorem 3. Let g G C'°°((0,1)) n (^([0,1]), then 

E45(e(t))] =E[5(0] +e-^i*E[<7(0Qi(e)]A(x) 

OO 

+ Y,e-^-\9:Qn)Pn{x), (27) 


where 


Qi(0 = 


n=2 


If g-P 
pi 1-p 


-1 


^>P 




(28) 


the left eigenfunctions times the stationary density when n > 2 are hyper¬ 
functions 

<3n(6 = ^^-5(-)(e-p); 


ni 


(29) 


and (^ — p) is the nth derivative, in a hyperfunction sense, of the Dirac 
delta function. Explicitly 


( 5 ) Qn) — 


j,W(p) 


ni 


There is biorthogonality of the right and left eigenvectors 

{Pmi Qn) — ^mw 

Proof. The left side of ([27]) evaluates to 

Ex[5(?(i))] = e~^9{qi{t-,x)) 

+ T] / 9iPii{x))qi{t - T;x)e ^dr. 


(30) 


(31) 


(32) 


*=1-^0 


If g has a power series expansion in (0,1) with a radius of convergence 1, 
then the first term on the right side of (1321) is equal to 


o— i 


= e~^g{p) + e~^ g‘^^\p)e~^^^ Pi{x) 

Pn{x) - CnlPlix) - CnO 




n=2 


ni 


(33) 
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because 


g{qi{t-x)) = g{p+{x-p)e 

The second two terms on the right of (I32p contain only linear terms in x — p. 
Now considering the right side of (|271) . (f33]l . and equating coefficients of 
Pn{x), n > 2, it must be that 


{g,Qn) = ^-^. (34) 

n! 

A property of integration by parts in the hyperfunction integral is 

= i-ir{g^^\5p) = (-l)V")(p), 

so ()34p is satisfied. There still needs to be a check on what happens with 
the terms Pi{x) and the constants in (j32h and (j33h . Consider the two terms 
in the sum on the right of (I32p . A calculation shows that the sum is 


+ 


g{p+{l—p)e (p + (x — p)e "^dr 


pe-^^/^) {l-p-{x- p)e-^^^-^^/^)e-'^dT 


[ 9{{P 

Jo 


= (l-e ^)g{p) + {x - p)g^^\p)e - e *) 

°° hn), 


n=2 


n\ 

n=2 

Adding (|33P and (1351) the right side of (j3^ is equal to 
9{p) - ^ - —i^Cno + ^ 

n=2 n=2 


g^'^\p) 

n! 


Cnl 


} 




n=2 


n\ 


(35) 


(36) 

(37) 
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To check the linear terms (j36p with the left side of (j32p consider 

i 

+- 


(1 _p)(_p)ri+p(l_p)r 

dyp) + 7 X — i - 

6 n\ 

n=l 

dip) - 2^ 


1 + n6»/2 


(38) 


n=2 


and 


E[ 5 ( 0 Qi(e)] 


^[{9{p) + 9(0 - 9 {p))Qi{0] 

0 + E[(5f(0 - 5 (p))<3i(0] 

r 

+^(1 “ J ( 5(0 - 9{p)) - pfl^~'^di 

2^ gW(p) (l-p)n-(-^)n 

n! ' l + (n-1)0/2 

n=l ^ ' ' 


,(i) 


ip)-Y. 


00 gin) 


(p) 


n=2 


n\ 


^nl • 


(39) 


Therefore (l36P is equal to 

E[5(0]+e"^^‘E[5(0Qi(0]A(x) 

showing that the left and right sides of (13211 agree. Checking the biorthogo¬ 
nality (fHTp . for n > 2, m > 1, 

(— 1 )^ 1 

{Pm,Qn) = = -,{P^\dp) = 5^^- 

n\ ^ n! 

For orthogonality of Pm, m > 2 with the first function Qi let g = Pm in 
(13^ . then 

E[T’m(0<3l(0] = Pm\p) - Cml^P^Hp) = Cml “ C^l = 0. 

'■ ■' ml 

□ 
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3. Non-mutant ancestral lines 


Let A^{t) be the number of non-mutant ancestral lines at time t back in 
a sample of n individuals taken at a current time. In the Kingman coales- 
cent {A^{t)}t>o is a simple death process, but in the A-coalescent multiple 
deaths can occur because of multiple mergers. Another difference is if the 
A-coalescent does not come down from infinity, then A^{t) = oo for t > 0. 
The death rates in the star-shaped coalescent are for i > 1 and j ^ i 


iO (2 j = i — 
1 j = l 


A similar argument to that in Section [2] is used to obtain the distribution of 
A® (t). We omit details of the calculation. 


Theorem 4. Denote the probability of no mutation along a line of length t 
by p{t) = t > 0 and let T be the time to coalescence in the population. 

Initially A® (0) = n. Then 


P{K{t)=j) 
P{K{t) = 1) 


p{K{t) = 0) 


P{Ai{t)=j,T>t) 

l^^p{t)\l-p{t)T-^e-\ 1 < j < n; (40) 

P{Ai{t) = 1,T > t) + P{Ai{t) = l,T<t) 

np{t){l - p{t))"'~^e~^ + J (l - (I -p(r))"-)p(t - T)e“^dr 

1 - (l - (1-p(t))”)e“* - [ {l-{l-p{T))"')p{t-T)e~'^dT 

Jo 

^ l + {k-l)9/2 • 


In the limit as n ^ oo the proportion of non-mutant lines at time t converges 
to p{t) if T > t; if T < t then there is a single non-mutant line if there is 
no mutation on the line between T and t or no non-mutant lines if there is 
mutation between T and t. Formally 


lim P{Ai{t) = I) 
1^00 

lim P{A^n{i) = 0) 


,-et /2 _ -t 


1 - 


1 - 0/2 ’ 
g-0i/2 _ g-t 


1 - 9/2 
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and 


lim P{A^{t) > 2) = e 

n—yoo 


—t 


with lim P{A^{t) 

n^oo 


k) = 0, k < oo. 


The loss of mass in the last equation is consistent with the star-shaped 
coalescent not coming down from infinity. 

Let be the time until there are no non-mutant lines. Then P(T® > 
t) = > 0). The mean of T® is 

poo 

E[T^] = / P{T^>t)dt 

Jo 

poo 

= / {l-{l-p{t)r)e-^dt 

Jo 

poo pt 

+ / (l - (1-p(r))"')p(t — T)e“’'drdt 

Jo Jo 

= (l + 2/0)(l-n!(l + 2/0)-J)), (43) 

after evaluation of the integrals. Notation is that a(„) := a(a-|-l) • • • (a-|-n—1) 
for a G M and n a non-negative integer. 

There is a duality relationship between ^{t) and Af^[t). Consider the 
probability that all individuals in a sample of n taken at time t are of type 
1. Knowing that at least one mutation has occurred on a line, the probability 
that it is of type 1 at forward time t is p. If T > t there are Af^{t) lines at 
time t back where the origin is with a frequency of x, then the probability 
of all being of type 1 is If T < t and Af^{t) = 1 then the single line 

at the origin must be of type 1 with probability x and the mutant lines at 
time T must be of type 1 with probability . If T < t and A^{t) = 0, 

then the mutant lines at T and the mutant single line from 0 to T must be 
type I with probability p ■ Therefore 

> t] 

= o,r < t}] 

= l,r < t}]. (44) 

Letting t ^ oo in (|4^ yields 

^lim E,,[^(t)’"] =p”+^E[(I/p)"^-(^)]. (45) 

The right side of (|45p provides a generating function for the number of 
ancestral lines at the time of coalescence, since we know the nth moment in 
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the stationary distribution on the left side, from (I38p . to be 


k=2 ^ ^ 


3.1. Spectral decomposition of P{Af^{t) = j) 


In the Kingman coalescent the spec tral decomposi t ion of the pr obability 
distribution of Af^(t) is well known ( Griffiths . 19801 : Tavare . 1984l h There 
is also such a decomposition for the line distribution in the star-shaped 
coalescent. 


Theorem 5. For n = 1,...; j = 0,..., n, 


n 

P{Ai{t)=j) = (46) 

k=0 

where Aq = 0, Ai = 9/2, Afc = 1 + k6/2, k > 2] Pq^'^ = 1, = —1, 

Qn^ = 1, n > 1; and for n > 1 


p(^) 

p{k) 

p{k) 



l + (i-1)0/2’ 


fn\ fc_i (fc-l)(l-kfc0/2) 
\kj^ ^ 1 +(A;-1)0/2 


0/2 

l-kA:0/2’ 


2<k<n-, 


1 < k <n-, 


1, 1 < A: < n; 

(k\ ,_i 1 +(A-1)0/2 

\jr ^ {k-i){i + ke/ 2 y 


2<j<n. 


(47) 


Proof. The decomposition comes from identifying the coefficients of powers 
of p{t) in (j4ni41l42|] . since = e~^p{ty, k > 2. also appears in 

(14T]l and ()42]l . The expansion ([46l) is upper triangular if 2 < j < n, but 
contains a non-zero term when A = 0 if j = 1. If j > 2, from m, 






(_i)fc-. 
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If j = 1, from dH]), 




n — 1 
k-1 


(-1)^-1- - - 

kj^ ^ l + (k-1)0/2 


, k > 1 


n\ fc_i (fc-l)(l + fc6>/2) 
kj^ ’ l + {k-1)9/2 


and if j =0 


«'-pr = E(;)(-i)-‘r 
2 = 1 ^ ^ 


1 


+ {i-1)9/2' 


-‘0 ~ 


n\ jk-1)9/2 ^ 

k)^ ^ 1 +(A:-1)0/2’ 


1 


«>d‘> = -E r; (-ir'i 


2 = 1 


+ {i-1)9/2' 


The definitions of and in ()17|) correctly satisfy these products. □ 


yiP u 


3.2. Population replacement 

Forward in time the times at which the population is replaced by the off¬ 
spring of a single individual form a renewal process with indepen¬ 

dent exponential increments Sj = Tj — Tj-i, j > 0 (Tq = 0). We find the 
density of type 1 individuals when k population replacements have taken 
place and express the transition density f{^]x,t) as a sum of these densi¬ 
ties. 


Theorem 6. Let fk{C‘,x,t) be the joint probability that there are k > 1 
population replacements in (0, t) and the density of the frequency of type 1 
individuals at time t, when initially the frequency is x. Then 


fk{i;x,t) 

where 

n(^T) 


+ {t^/k\)e \ i > p + [1 - p)e 

) {k-i 

1^1 +I log ( 2 ^) j {t^/k\)e-\ e < p(l - 

(48) 

p + {x - p)e~^^P\ — r 2 {^;t) = l- p- [x -p)e~‘^^P-^. 

i-p p-i 
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Proof. Let P{s) = be a transition matrix for the probability of 

a replacement of a population of type i individuals by type j individuals at 
time s. The entries of P{s) are those in Theorem [TJ That is 


P{s) 


p+{l—p)e (1—p)(l —e 

p(l _ g-6's/2) X _ p _|_ pg-^'s/2 


The left and right eigenvector matrices of P{s) are 


U = 


p 1 — p 
1 -1 


, u-^ = 


with eigenvalues l,e Therefore 


Pis) = U-^Biag{l,e-^^/^)U, n^(si) 

i=2 


1 1 — p 

1 —p 

C/-^Diag(l,e-®(‘'=-‘i)/2)l7, 


where the product is taken from P{si) on the left to P{sk) on the right. At 
the population replacement is of type 1 with probability qi{ti;x). Thus 
the probability of type 1 or 2 replacements at time conditional on {tj}j^^ 
is 




[qiih-,x), 1 - gi(ti;x)] ]JP(sj) 


i=2 


[p+ {x — p)e T®/ 2 ^ 1 _ p _ ^ 2 ; _ p^g tfc^'/ 2 j ^ 


(49) 


Then the conditional frequency of type 1 individuals at time t is 
Piiit — tf:) with probability ri(x; {tj}j^i) 

P 2 iit — tk) with probability r 2 (x; {tj}^^^), (50) 

where R{x‘, (a;; T 2 (a^{ij}j=i)] • The probabilities in 

(l50]l only depend on the time tk- Tk has a probability distribution, condi¬ 
tional on k replacements in (0,t), of 

k{tk/t)^~^t~^, 0 <tk <t. (51) 


A similar calculation to that in Theorem [T] gives that the probability of 
having k > 1 replacements in (0,t) and density of ^(t) is (I48p . There is an 
expansion of the transition density 


fiC]x,t) = '^fkiC;x,t). (52) 

k=l 

It must be that (f5^ holds and a check shows this to be true. □ 
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4. The star-shaped coalescent with d types 

In this section an extension is made to d types, then the infinitely-many- 
sites limit as d —)• oo is taken. The model is a star-shaped coalescent with 
mutation occurring on the lineages at rate 9/2, with the type chosen as i 
with probability pi = 9i/9, i = 1, ... ,d. Analysis of the model in Theorem 
[3 follows that when d = 2 in Section [2j The proof is omitted because of 
similarity with Theorem [TJ 

Theorem 7. Let {^{t)}t>o be the frequency of d types of individuals at 
time t, when ^(0) = x, in the Fleming-Viot process with a generator acting 
on functions g{x) E C'^(Ad), = {a;; xi = 1}, of 


d 


d 


^9{x) = '^Xi[g{ei) - g{x)) + {9/2) '^{pi - Xi)gi{x). (53) 


1 = 1 


Denote pij{t), i, j = 1,... d as the probability of a change of type in a single 
line of the star-shaped coalescent forward in time from i to j in time t and 
qi{t-,x) as the probability a single gene chosen at time t forward in time is 
of type i = 1,..., d. Specifically 


Vij{t) = Sije + {l-e 
qi{t; x) = + (1 - 


Then 



qr(t; x) with probability e * 

Pi{T) with probability (1 — e~^)qi{t — t;x), i = 1,... ,d, 


(54) 


where Pi{T) is the ith row of P{t) = {Pij{T))'l and r has a truncated 
exponential density The continuous part of the distribution of ^(t) has 
a density in the zth region for ^j(t) of 



ii > Pi+ {l-pi)e 

(55) 


and 



The stationary density in the rth region is 



(56) 
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and 



Remark 2. There is a similar representation to that in Remark [T] for the 
stationary distribution ()56p of Let t/ be a random variable with density 
, 0 < r] < 1. Then for i = 1,..., d, with probability pi, 


Ci = (1 - v)Pi + V, 

j ( 57 ) 


Remark 3. The random variables {^j} are exchangeable if pi = 1/d, i = 
1,... ,d with one taking the value {1 — r])/d + ri and the d — 1 other random 
variables taking the value (1 — r])/d. In the limit as d —)• oo there is one 
frequency of r] and an infinite number of dust frequencies of total mass l — r]. 
The limit sampling distribution in n unlabelled individuals is 



j! ■ (l + 2/d)(„)’ 


(58) 


where there is one sampling frequency of j and n—j distinct frequencies of 1 . 
Straightforwardly there is one type with probability E [ry”] and k types with 
probability Thi s samp ling distribution agrees 

with the sampling distribution (27) in iMohlel (2006|) after simplihcation and 
identification of parameters. 


4.0.1. A general mutation model 

A similar theorem to Theorem [7] holds for the general Markov mutation 
model without the exact detail of the exact transition density and stationary 
density. The mutation model is that mutations occur along lines at rate 0/2 
and, forward in time changes of type are made from i to j according to 
a transition probabiity matrix P. It is assumed that P is recurrent with 
stationary distribution 7 . 

Theorem 8 . Let {^{t)}t>o be the frequency of d types of individuals at 
time t, when ^(0) = x, in the Fleming-Viot process with a generator acting 
on functions g{x) G C'^(Arf), = {x; xi + • • • + = 1}, of 

d d d 

^9{x) = - 9{x)) + {0/2) ^ “ Xi)'^9i{x). (59) 

2=1 2 = 1 j = l 
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Denote Pij(t), i,j = 1,... d as the probability of a change of type in a single 
line of the star-shaped coalescent forward in time from i to j in time t and 
qi{t;x) as the probability a single gene chosen at time t forward in time is 
of type i = 1,... ,d. Specihcally 


d 

P{t) = exp{(6l/2)(P - I)t} , qi{t\x) = '^XkPki{t). 

k=l 


Then 




q{t;x) 


with probability e * 

with probability (1 — e~^)qi{t 


i • 1 W 

r;®), ^ = 


where Pi{T) is the ith row of P{t) and r has a truncated exponential density 
(j9|l. If ^ is a random variable with the stationary distribution of {${t)}t>o, 
then 

^ Piir) with probability 7 *, i = 1,... ,d (61) 

where r has an exponential distribution of rate 1 . 


5. The star-shaped coalescent with general frequency dependent change 

Consider a population of individuals of types 1 or 2 evolving in time and 
let ^(t) be the frequency of type 1 individuals at time t, when ^( 0 ) = x. 
is a Fleming-Viot process with with generator 

Cg{x) = x[g{l) - g{x)] + (1 - x) [^(0) - g{x)] + v{x)g'{x). (62) 

At time t after a replacement and before the next replacement the frequency 
of type 1 individuals is x(t) satisfying 

^Xit) = v{x{t)) (63) 

with x( 0 ) = 1 or 0 depending on the type of the replacement being respec¬ 
tively 1 or 2 . {^{t)}t>o is a jump process such that when there is a jump 
the population is completely replaced by either all type 1 genes or all type 
2 genes. The time T between jumps has an exponential distribution with 
parameter 1. Between these replacements ^{t) changes deterministically 
governed by v{x). Typically the model is one with mutation and selection 
where 

1 1 
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with x(0) = 1 or 0 depending on the type of the replacement being respec¬ 
tively 1 or 2. 

For simplicity we restrict attention to two types and the stationary dis¬ 
tribution of {^(t)}t>o in this section, though there are also analogous results 
to the case with no selection when there are d types and there is also a 
solution for the transition functions of {^(t)}t>o- 

Theorem 9. Let /i(t) be the solution to (I63p when a type 1 replacement 
occurs with /r(0) = 1 and Ty{t) the solution when a type 2 replacement occurs 
with z^(0) = 0. Let P = {pij) be the transition matrix between jumps of 
type i,j G {1,2}. Then 


/E[M(r)] E[i-^(r)]\ 

Vekt)] E[i-z.(r)];- 


(65) 


T is exponential with rate 1, and P describes then skeleton transition matrix 
between the states of the population at instants after replacement when the 
population is composed of just one type. The stationary distribution of the 
type of replacement (vri, 7r2) is the stationary distribution (tti, 7 r2) of P, that 


IS 


EHT)] 


^ E[u{T) + 1 - p{T)Y ^ ^ 

The stationary distribution of {C(^)}t>o is the distribution of /r(r) if the 
last replacement was type 1 or i/(r) it was type 2, where r is an exponen¬ 
tial random variable of rate 1. A random variable ^ with this stationary 
distribution has a density 


f{Y,v) = TTie ^ 


HO. 


1 




+ vr2e 


HS). 


1 




(67) 


The mean E [^] = tti . 

Proof. The frequency of type 1 individuals at the instant before replacement 
is either fi{T) or v{T) depending on whether the prior replacement was type 
1 or 2. A choice of a type 1 or 2 replacement is then either E[|u(r)] or 
E[i/(T)] which is the first column of P. The second column follows similarly. 
The stationary distribution is clearly (7ri,7r2). The times of replacement 
form a renewal process with exponentially distributed increments T of rate 
1. The asymptotic distribution of the time from the last replacement to a 
given time point between two replacements which cover the point is again 
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exponential r of rate 1. Therefore a random variable ^ with the stationary 
distribution satishes 


IK't) 

The density is then (j67p after a change of variable r —The inverse 
functions and may have disjoint support regions, such as in 

(HU). The mean of ^ being tti follows from (|68p . □ 

5.1. Fixation probabilities in a model with selection 

If there is no stationary distribution then the probability of fixation of type 
1 individuals is the probability that the first jump is of type 1. Let Pi{x) be 
the probability of fixation beginning at x. Consider the particular selective 
model where there is no mutation and v{x) = (/3/2)x(l — x). Then 

Pi{x)=E[xiT)], (69) 

where x satishes (f63P with x(0) = x. The logistic curve solution to ([631) is 

^ (1 — + X 

and 

/■oo /■! 

PiW = y^ e-\(t)dt = i?/fi)^I l_(l_^)(l_^) dz. (70) 

The probability hxation of type 2 individuals, from an initial frequency of 
y = 1 — X is, from ([70P . 

PM = 1 - (2//3)(l - y) r dz. (71) 

Jo l-2/(l-^) 

5.2. Mutation and selection 

If there is mutation between types 1 and 2 then a stationary distribution 
exists. 

Theorem 10. In a model with mutation and selection 
^^(a;) = - Ox) + ^/3x(l - x). 


with probability vri, 
with probability 1 ^ 2 . 


( 68 ) 
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where 0 > 0, /3 > 0. Let 0 < p = 6i/9 <!,(/> = 9//3 and 0 < ri < 1, r 2 < 0 
be the two (real) roots of 

X(1 - x) + - X) = 0, (72) 


specihcally 

ri,r2 = ^(l-(i)± V(1 - (/>)2 + 44>p'^. 

Then the probability of replacement transitions at time t, P{t), has elements 


= p{t) 

P2i{t) = fit) 


lull g-/3(»'i-r’2)t/2 

- (j'-i — T’o')- 1 — ^ - 

1 l-n „-/3fr-i-r?H/ 


^ 1 _ i:iIig-/3(ri-r2)t/2 

1—r2 

ri „-3(ri -ri)tl2 


(73) 


and P = E[P(T)] . The stationary density (1671) is 



2 



^-r2 

n - ^ 
^-r2 


1 ^ 1 
l-nj (^ - r 2)2 


y^/{ri < ^ < 1} 

1 - ri 


2 1 

^ 1 

n y (n - 0^ 


^/{o < C < n}. 
n 


(74) 


Proof. The differential equation (IMl) for x(t) is 


^X=^[0{p-x) + PxCi-- X))- (75) 

To obtain the solution of (|75]) let 4> = 9//3 and factorize the quadratic term 

Q{x) ■=-X^ + {I - (I^)X + Pf)- (76) 

Let ri, r 2 be the roots of (I72]l in the statement of the theorem. <5(0) = pcj) > 
0 , ( 5 ( 1 ) = —(1 — p)f> < 0 and Q{x) “Oo as x —>■ —oo so 0 < ri < 1 and 
r 2 < 0 . ([75]) can be written as 

_ = 

ri — r2^X — X — f 2 ’ dt 2 

Therefore 

- x{t) ^ n - x(0) g(n-r2W2 .77^ 

X(t)-r 2 x( 0 )-r 2 ■ ^ ^ 
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Let 


C'(x(0);t) 


x(0) ^-/3(ri-r2)t/2 

x(0) - r2 


The solution of (|75l) can be written as 


X{t) = ri- (n - r2) 


g(x(o);t) 

1 + C'(x(0); t) 


( 78 ) 


Now ^(t) = xit) when x(0) = 1 and u{t) = x{t) when x(0) = 0; which gives 
([73]). Note that (7(1; t) < 0 and C{0;t) > 0. To find the stationary density 
make a change of variable t —^ where r is exponential with rate 1 and 
^ = x{t)- The density, with x(0) either 1 or 0, is 

ri-x(o)y (^-?’2)|?-nr 


where the support of the distribution depends of the positivity of the ex¬ 
pression in large brackets. Evaluating (17^ in the two cases gives (I74]l . 


□ 


Corollary 2. Let o = (ri — r2)/2 > 0, 0 < 6 = (1 — ri)/(l — r 2 ) < 1, and 
c = —r\jr 2 > 0. Then 


Pii = 


ri + - y)-^dy 

7o 

°° Uk+l 

ri +2/3-1^ 


( 80 ) 


k=0 


l/(a/3) + A: + 1 

c/(l+c) 


P21 = 


n - r " - y) 

Jo 


_,A-l/iaP)-l^y (81) 

1 / {a0)-\-k-\-l 


^ ^ ^ kl l/{al5) + k + l 


TTi and 7r2 can be computed from pn and p 2 i- 

Proof. The inequalities for a, b, c are elementary. Let r be an exponential 
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random variable with rate 1 . 


pii = E[pii(r)] 

P 21 = E [p 2 i (r)] 

pco 

= ri — 2a e~^ - w^dt. (83) 

io l + ce-“/3i ^ ^ 

Now make the changes of variable y = in (|82p to obtain (j80p and 

y = ce““^*/(l + ce““^*) in ([83l) to obtain ([Hip . □ 


In a model with no selection with /I = 0, pu = p 2 i = p- Letting /3 ^ 0 
gives ri p, r 2 ^ —oo, r 2/3 ^ —0 with ([73]), (|80l) and ([HTI) converging to 
the correct neutral quantities. 


5.3. Ancestral selection graph 
5.3.1. Duality 


There is an ancestral selection graph which is dual to a process 
with a generator described by 

£g(x) = x{g{l) - g{x)) + (1 - x) {g{0) - g{x)) - ^x(l - (84) 

where /? > 0 and 5 G C'^([0,1]). (If /I < 0 then consider the process {1 — 
For i G Z+ 


/:x* = x-x* + ^(x*+i-x*). (85) 

Re-interpreting (l 8 ^ as a generator acting on i, the dual process {Bn{t)}t>o 
with BniO) = n is a Markov process on Z+ with rates 



j = i + l, 

j = 1- 


The duality equation is, for n G Z+, 

E4^(t)"] =E„[x^"W], 


( 86 ) 


(87) 


where expectation is on the left with respect to ^(t) and on the right with 
respect to Bn{t). {Bn{t)}^^i does not describe the full ancestral selection 
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graph, but it is a lineage count ing process. Duality is studied in the Wright- 
Fi sher model with select ion in Mano ( 20091 1 and in a A-Fleming-Viot model 
Etheridge et ah ( 2O10l l. Duality is much simpler here and we have explicit 


m 


results. 


5.3.2. Stationary distribution 

There is a stationary distribution tt for the edges in the ancestral selection 
graph calculated from the rates (l5St) which satisfies 


vr,; = 


(2//3 + l)(j) 


VTi, i > 1. 


TTi = 2//3 is evaluated from 

OO 

1 = E 


TT,: 


2 = 1 


r(i)r(2//3 + 1 ) 


OO 

= TTi ^ — z)'^^^dz 




TTi / (1 - zf/^-^dz 

Jo 

(^/2)7ri. 


Therefore 


VTi = 




( 88 ) 


{ 2/13 + l)(j) 

Letting t —)• oo in the duality equation (IHTl) 

E,[^(oo)"] =E4y^"(-)] =P2{y), 

taking into account the negative selection coefficient in (|84D . Therefore a 
generating function for Bn{oo), which has a stationary distribution 
is 

OO 

= P2{y). 

2=1 

Calculating the coefficie nt of y'^ in Poi v) from (f7T]l shows this to be true. 
This argument is used in iManol (120091 1 for the Wright-Fisher diffusion with 
selection. 
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5.3.3. Time to the ultimate ancestor 


The ultimate ancestor is reached when Bn{T\jp^) = 1 for the first time at a 
random time Tua- The ultimate ancestor is reached with probability one, 
because the probability that Bn{t) > 1 for all t > 0 is 



In fact the ultimate ancestor is always the most recent common ancestor in 
this star-shaped coalescent unlike in the Wright-Fisher model with selection 


where the event depends on the type of the ultimate ancestor. Let 
be the Laplace transform of the time to the ultimate ancestor Tua from 


-Bn(O) = n individuals. It must be that = (1 -|- <y9) n > 2. We give 

a formal proof of this. {Gn{<^)}n >2 satisfies the system of equations 



n/3/2 


1 


Gn+i{^),n>2 (89) 



ip + l + n/3/2 


with Gi{(p) = 1, from a decomposition according to the first jump. Rewrite 
and extend (f5U|l for > n as 


n/3/2 


((1 -I- ^p)Gn+i{^) — l) 


(1 -|- tf)Gn{^) — 1 


ip \ nfi / 2 



Now let iV —>• oo in ([90]l . (1 -|- (p)G ^— 1 is bounded and 



so Gn{ip) = (! + (/?) ^ 
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